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ABSTRACT 

The Sloan Pillars have set the standard for university-wide online learning program assessment for more 
than a dozen years. In this paper, the authors propose the extension of the Pillars to corporate e-leaming, 
offering an alternative to traditional enterprise learning assessments. Claiming that conventional methods 
stress individual courses or programs, rather than encompassing a company’s learning ecology, a new 
corporate version of Sloan Pillars is proposed. The authors claim that their holistic approach — assessing 
employee access, learning and cost effectiveness, and learner and management satisfaction — provides 
companies with the tools they need to assess the effectiveness of company learning efforts overall. 
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I. INTRODUCTION 

In large corporations, education and training are recognized as necessary functions, hut their value in 
many companies is fraught with ambiguity. As a rule, corporate management tends to he unclear about 
what outcomes are required. For some, training is introduced to support worker retention; in which case, 
education programs are considered an employee benefit falling under the arm of human resources. 
Alternatively, training may be called upon to improve job performance or enhance technical skills or 
specific training requirements may be imposed through government mandates; under these conditions, a 
company’s training organization commonly manages the learning enterprise. These examples are merely 
two bracketing a much larger range of possibilities. 

In the current economic climate of layoffs and restructurings, learning executives have never felt a more 
urgent need to rationalize or justify the value of education and training in advancing company interests. 
More than ever, corporate learning officers are now eager to demonstrate that their organization generates 
rapid and quantitative corporate benefits. For many, the wish to prove the value of corporate learning 
opens a search for an industry "standard,” one that effectively measures the efficacy of learning programs. 
If outcomes are measured against a standard template to reveal quantitative results, and if outcomes are 
positive, learning departments say that their efforts deliver productive programs — or so the thinking goes. 

It so happens that a widely practiced industry "standard" does exist — the Kirkpatrick method, originally 
proposed by Donald Kirkpatrick and first published in a series of articles in the US Training and 
Development Journal in 1959 [1]. It has been modified and updated many times since. 
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To evaluate the effectiveness of a learning program, the Kirkpatrick model outlines 4 levels of assessed 
outcomes: 

Level 1: Reaction-Did employees like the learning experience? 

Level 2: Learning-Did they ahsorh what they were taught? 

Level 3: Behavior-Did they use what they were taught? 

Level 4: Results-'^da.t is the impact of what they were taught? 

While a complete program evaluation requires measuring all 4 Kirkpatrick levels, a valuable attrihute is 
that each level provides useful, if partial, input towards overall program evaluation, even if all levels are 
not, or cannot, he measured. Kirkpatrick’ s method also has the advantage of being relatively intuitive and 
easy to understand. At a quick perusal, it seems quite logical and sensible. After all, who can reasonably 
oppose measuring a straightforward objective, such as, "Did they learn what they were taught?” 

Originally, the Kirkpatrick model was introduced as a way to assess effectiveness of learning in 
manufacturing and similar industries, contexts widespread in companies of the day that promoted the 
“right way” of doing things. Successful training urged learners to do things the “right way.” Modern 
versions of the old manufacturing-style training continue to be offered today, such as courses in basic 
quality principles (Six Sigma, for example), or training in the use of a new release of computer-aided 
design (CAD) software for engineers in a computer-chip facility. Yet another might provide instruction in 
the most efficient and safe ways of packing widgets coming off a production line. An assumption 
underlying Kirkpatrick’s model is that what is presently taught is appropriate for current needs of the 
organization. 

In the absence of other widely established standards, the model has been broadly adopted and extended by 
corporations and promoted by consultants as a gold standard against which to measure effectiveness of 
learning across an organization, including teaching complex behavioral, leadership and problem-solving 
skills — skills required at professional levels in today's complex, continuously changing business 
environment. Because the Kirkpatrick model assumes direct applicability between what is taught to what 
it is applied, it may not be accepted as an accurate measure by executives in assessing skills required in 
work environments in which the classroom version may be quite different from what is practiced on the 
job. 

The Kirkpatrick model was designed primarily to examine the effectiveness of individual training 
programs, with the assumption that if you measure isolated Kirkpatrick outcomes of different courses one 
by one, you should come up with a good assessment of learning effectiveness and corollary objectives 
(such as cost effectiveness) across your entire organization, an assumption that needs to be examined. 


II. HOLISTIC APPROACH 

The Kirkpatrick model is useful and appropriate for the many training programs. Nonetheless, there is a 
need for a more holistic approach to assess learning across organizations, including measuring the impact 
on a company’ s overall learning environment. Such a holistic approach needs to account for possibilities 
not foreseen when Kirkpatrick first conceptualized his model, such as the penetration of e-learning in 
corporate training. 

While departments offering targeted courses are understandably interested in the impact of their own 
programs, still, individual programs need to be evaluated in the context of the organization as a whole. 
Instead of using a simple average or aggregation of Kirkpatrick results from all programs delivered, 
management needs to find a way of measuring the performance of the learning enterprise across the entire 
corporation. The effort must include understanding how separate programs interact with one another as 
well as how learning units perform in the environment in which the company does business. It must as 
well, include baseline assessment in order to establish a credible claim for improvements originating from 
a learning program. 
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In developing a more holistic approach, a few criteria need to he kept in mind. Successful assessment 
models, not only those covering training, usually feature simple, easy-to-rememher factors, with easy-to- 
understand metrics and fairly limited accompanying data requirements. The Kirkpatrick model, whatever 
its shortcomings, is simple and intuitive. Verifying data must he easy to collect and analysis following 
data gathering must not only account for variables, hut must he easy to understand. 

With each "improvement" cycle over time, the original simplicity of the Kirkpatrick model has evolved 
into a highly complex set of variables in which statisticians are now called upon to collect and analyze 
datasets. Some, for example, integrate Level 4 results with financial records in an attempt to measure 
return on investment (ROI). The model is even employed in programs that deal with complex leadership 
and sales behaviors. Typically in these analyses, business-environment variables, such as the impact of 
marketing and seasonal variations, are held constant or, even worse, ignored. Routinely, resulting detailed 
complex analyses lead to derivations of ROI results that are often amazingly impressive but also 
frequently non-credible. {Also see article “Measuring Success and ROI in Corporate Training” by Kent 
Barnett and John R. Maddox, II, in this issue.) 


III. COMPLEMENTARY APPROACH: SLOAN-C PILLARS 

In parallel with widely adopted technology-based methods of delivering corporate training, universities 
have been equally active introducing e-leaming [2]. Reliable estimates indicate that today more than 5 
million students are enrolled in degree-granting institutions across the US, taking at least one course 
entirely online. [3] These students represent approximately 25% of all those enrolled in American degree- 
granting institutions. The Sloan Consortium (Sloan-C), the professional society of institutions and 
individuals practicing e-leaming, with special emphasis on colleges and universities, has promoted a 
different approach to assessment of quality education — Sloan-C Pillars. 

Sloan-C Pillars recognize that much of organized education today is connected in some way to e-leaming. 
Sloan-C Pillars also take into account that e-leaming and other electronic methods must earn their way 
into widespread deployment by demonstrating their advantages to new learners through learning 
effectiveness, cost-effectiveness and learner satisfaction. In this paper, we propose the possible 
application of Sloan-C Pillars to corporate learning [4]. 

The 5 Sloan-C Pillars were originally devised to answer questions that must occur to every leader who 
manages online offerings in a learning organization: How do I know I am operating a quality 
organization and how do I continually improve it? 

Four of the 5 Pillars were first presented at an academic conference in 1997 [5] and subsequently 
described in an interview published in 1998 [6]. The fifth Pillar — overall student satisfaction — was added 
a year later. The idea behind the Pillars is that a quality organization must do more than ensure that 
intended learning outcomes are achieved; it must also demonstrate success along 4 separate but 
interrelated dimensions. The original Pillars were intended for college and universities. In this paper, they 
are referred to as “Pillars A.” The 5 elements in Pillars A are: 

1. Access 

2. Learning effectiveness 

3. Cost effectiveness [7] 

4. Faculty satisfaction 

5. Student satisfaction 

As already noted, a successful quality organization with ambitions to achieve continuous improvement 
must recognize that it cannot focus on learning outcomes only, as important as they are, but must assess 
itself along multidimensional axes, or “Pillars,” on which the organization is metaphorically supported 
(Figure 1). Because each element in Pillars A is equally important in supporting the structure, none can be 
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eliminated. While Pillars A represents a different approach than Kirkpatrick’s model, the two are not 
directly opposed. Rather, they can he thought of as complementary. Embedded in the Pillars are elements 
of the Kirkpatrick model. 
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The Sloan-C Pillar Approach to Evaluating Learning 


Figure 1. Pillars A 


IV. EMPLOYING SLOAN-C PILLARS 

Let us now explore in more detail what the Pillars represent and what metrics might he used to evaluate 
performance of a learning organization against each pillar. 


Access refers to the number of people engaged by learning programs. In organizations with 
substantial e-leaming components, it is reasonable to ask why digital options are introduced at all. 
Given the extra costs that online courses require, if a significant new population, over and above 
those reached in traditional classrooms, is not added, why bother? How many new learners are 
being reached, and is that number sufficient to justify the technology expense? Access must also 
constructively address situations where there are multiple access points for learning programs 
within the corporation, such as employees who wish to access through mobile platforms. 

Learning effectiveness is a way of inquiring whether learning objectives sought are met. Because 
“second rate” learning is intolerable, the criterion for success must be — “results equivalent or 
better than might be achieved in classrooms alone.” It is quite possible that “better” is the more 
likely outcome when e-leaming is deployed. Properly constmcted, e-learning can do more for 
learner engagement than old-style classroom learning, but appropriate metrics need to be 
constmcted to demonstrate that is in fact the case. 
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Cost effectiveness must also he carefully assessed to ensure overall costs are contained enough to 
enable programs to scale-up for widespread deployment. If merely experimental, small-scale 
learning programs are sustainable, or if learning outcomes deteriorate as you scale-up, the 
program is unacceptable. 

Since virtually all online courses at colleges and universities are faculty-led, whether online or 
blended, it is critical to assess whether faculty are satisfied with their role. Under faculty 
satisfaction, if online instructors are not attracted to e-leaming, it is a clear mark of failure. 

Finally, it is necessary to assess overall student satisfaction. Deteriorating or low rates of student 
satisfaction will result in a gradual decline in enrollment, signaling impending failure. 

If Sloan-C Pillars are introduced as benchmarking criteria and as methods to drive continuous 
improvement, then each pillar must measure success against a set of metrics. Benchmarking and 
quantitative measures of quality are not easily adopted by academic institutions, but there has been 
gradual progress with scattered examples of effective practices. In academic settings, metrics might be 
generated for each pillar along these lines: 


Access: University records can show geographical distribution and student enrollment. 

Learning effectiveness: Student grades, retention, graduation, postgraduate employment and other 
conventional methods of learning effectiveness can be recorded. 

Cost effectiveness: One can measure the relative cost of conventional versus e-leaming options. 

Faculty satisfaction: Faculty surveys and interviews, as well as the number of faculty 
volunteering to teach online, are good metrics. 

Overall student satisfaction: Course satisfaction surveys are commonly used at universities and 
are a good metric. Student retention or course completion rates are also relevant and can be 
applied to learning effectiveness as well [8]. 

We now turn to the matter of applying the Pillars to corporate environments. As a first approximation, it 
appears that all the elements in Pillars A can apply quite nicely — with one exception — to corporate 
learning. That exception is faculty satisfaction. This Pillar, so natural in academic settings, does not fit 
industry at all. In corporations, instructor-led online learning represents only a small part of training and 
will doubtless remain so for quite some time. Nor is instructor satisfaction in corporate learning a critical 
dimension. 

Because management satisfaction is absolutely critical for corporate learning to succeed, we have 
replaced /acw/ty satisfaction with management satisfaction as an essential corporate pillar. Since learning 
organizations occupy cost-centers, not profit-centers, at most companies during economic downtowns, 
they are vulnerable and subject to cost-cutting. Consequently, the learning unit must devise appropriate 
metrics to capture management’s objectives. Used effectively and judiciously, metrics generated for 
management satisfaction can be a useful communication tool for the learning unit to engage management, 
helping to establish a clear understanding of what goals are expected. A version of the management 
satisfaction Pillar was introduced at Johnson & Johnson and Amway to assess improvements in 
organizational learning. 
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Let’s now take a look at how the model can he applied in companies. While we have chosen to look at the 
application of the Sloan model from the perspective of a for-profit organization, there are close parallels 
in other organizations as well. 


Access to learning addresses the question of reach of learning across the organization. Metrics 
might include the percentage learning programs are accessible to employees. If your company has 
introduced learning technology to deliver programs, such as a learning-management system 
(LMS) or a portal, you might also measure accessibility to technology across the organization. 

Learner satisfaction covers employee satisfaction with learning and personal growth 
opportunities. This pillar might include the Level 1 metrics from the Kirkpatrick model. What’s 
more, if your company surveys workers to learn their opinion on various staff services, the results 
might be used to determine whether employees’ perceived personal growth and development 
needs are being met. Other possible metrics might include an index of the frequency of use and 
completion rates. In order to produce the most favorable Learner Satisfaction metrics, it may be 
necessary to tailor learning differently for associates and for executives. Similar content might be 
offered in compressed, short form for executives, with longer more detailed version offered to 
associates. If the needs of each group are being met, substantially improved outcome metrics are 
likely. 

Cost effectiveness examines the capital efficiency of learning investments, with the objective of 
determining how much is accomplished with given resources and budgets. It is tempting to 
exploit the data to measure ROI, but there are other and better indicators. Furthermore, ROI of 
individual programs does not necessarily mean that capital was invested effectively within the 
wider range of corporate investments and returns. 

Learning effectiveness examines indicators showing the impact of learning on the organization’ s 
strategic direction. Some of Kirkpatrick’s Levels 2 and 3 might be adapted for use here. Cross- 
links might also be made with the organization’ s innovation or speed-to-market indicators 
coupled with employee perceptions of their growth and development. If a correlation analysis is 
employed, caution must be taken not to attribute cause and effect since correlations often examine 
connections but do not necessarily show casual direction. 

Management satisfaction looks at indicators of management satisfaction with learning introduced 
across the enterprise. An obvious indicator is the trend of learning budgets over time relative to 
business performance. Surveying management satisfaction with the learning enterprise might be 
another approach. A qualitative metric demonstrating the record of communication between 
management and the training unit might also be considered. Currently, many managements are 
also struggling with the issue of how to ensure sustainability for learning programs. The learning 
function needs to address this and devise appropriate metrics for management review. 

The modified corporate version of Sloan-C Pillars (Pillar B) is shown in Figure 2. 
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The Sloan Consortium’s Five Pillars Approach 
to Evaluating Learning 



Figure 2. Pillars B 


V. CALL TO ACTION 

The Pillars not only represent a straightforward assessment model that can he applied in both academic 
and corporate contexts, hut they are also a comprehensive and holistic approach designed to achieve a 
hroad-stroke picture of how well or poorly a learning organization is performing. The Kirkpatrick model 
and its variants are so embedded in the corporate learning community culture that it may be in a state of 
what behavioral psychologists call “inattentional blindness.” Rather than look at other, more 
comprehensive ways of measuring organizational learning, corporate learning executives may be content 
to continue to focus on assessing specific training programs, rather than the effect of learning on the 
enterprise as a whole. 

For the Sloan-C Pillars to be introduced widely as a holistic tool in corporations, learning professionals 
might usefully take ownership of it — as end-users do with open-source software — adapt it, improve it, 
and share it with the rest of the corporate learning community. We are open to exploring ways the Pillars 
might be applied in corporate environments, perhaps in discussions on the Sloan-C website at 
www.sloan-c.org, and in industry periodicals and conferences supported by ASTD and Sloan-C. 
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